{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 机器读心术之神经网络与深度学习第2课书面作业\n",
    "学号：207567\n",
    "\n",
    "**阅读作业:**  \n",
    "1. 精读《Understanding the difficulty of training deep feedforward neural networks》一文\n",
    "\n",
    "\n",
    "**书面作业：**  \n",
    "1. 根据《Understanding the difficulty of training deep feedforward neural networks》回答下列问题\n",
    "  * normalized的初始化权值和standard的初始化权值在定义上有何不同？\n",
    "  * 根据文中图11，比较一下几种激活函数的优劣，什么原因造成sigmoid效果比较差？而softsign效果相对较好？\n",
    "2. 用任何一种编程语言编程实现LMS学习算法，并将运行测试的结果抓图"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 书面作业第1题\n",
    "根据《Understanding the difficulty of training deep feedforward neural networks》回答下列问题\n",
    "  * normalized的初始化权值和standard的初始化权值在定义上有何不同？\n",
    "  * 根据文中图11，比较一下几种激活函数的优劣，什么原因造成sigmoid效果比较差？而softsign效果相对较好？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**答**:\n",
    "\n",
    "1. normalized的初始化权值和standard的初始化权值在定义上有何不同？  \n",
    "   \n",
    "**normalized initialization**来初始化权值定义如下：\n",
    "\n",
    "$$\n",
    "W^i \\sim U\\Big[-\\frac{\\sqrt{6}}{\\sqrt{n_i+n_{i+1}}},\\frac{\\sqrt{6}}{\\sqrt{n_i+n_{i+1}}}\\Big] \\tag{16}\n",
    "$$\n",
    "\n",
    "上式的方差：\n",
    "\n",
    "$$\n",
    "Var[W^i]=\\frac{2}{n_i+n_{i+1}}\\tag{12}\n",
    "$$\n",
    "\n",
    "根据论文中推导，这样的初始化在前向传递的值以及后向传递的梯度的方差会保持稳定。  \n",
    "\n",
    "**standard initialization**，也就是论文中的公式(1)，定义如下:\n",
    "\n",
    "$$\n",
    "W_{ij}\\sim U\\Big[-\\frac{1}{\\sqrt{n}},\\frac{1}{\\sqrt{n}} \\Big] \\tag{1}\n",
    "$$\n",
    "\n",
    "**standard initialization**下权值的方差，也很容易计算（假设所有层维度都是n）:\n",
    "$$\n",
    "Var[W]=\\frac{1}{3n}\\tag{15}\n",
    "$$\n",
    "\n",
    "根据式(15)，**standard initialization**会导致反向传递的梯度依赖于层的维度，同时可能会变小（饱和）。\n",
    "\n",
    "1. 根据文中图11，比较一下几种激活函数的优劣，什么原因造成sigmoid效果比较差？而softsign效果相对较好？\n",
    "   + Sigmoid效果比较差的原因有：\n",
    "     + sigmoid非0均值会诱导在Hessian矩阵中的重要奇异值导致训练慢   \n",
    "     + sigmoid函数在靠近渐近线时变化比较快，也就是说容易饱和； \n",
    "   + softsign在靠近渐近线时变化比较平缓，也就是不是很容易饱和。\n",
    "  我画下面一张图，比较了sigmoid、tanh和softsign函数，为了比较方便，我将sigmoid函数整体下移了0.5这样看起来更方便，可以看出这种趋势来。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x239892e7bb0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEICAYAAABS0fM3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABEa0lEQVR4nO3deXxU1fn48c+TycoiYYeELSAgaCRABBQXcAFFEHAruKFWqVJfUi0qtrZfWrWite2vLsG61aWCdamKiqK4gAQ3wi6QsAUkARIISyAJSSbn98edJJPJTDKTmckkmefNa14zc+52ckPOc++5ZxFjDEoppcJXRKgzoJRSKrQ0ECilVJjTQKCUUmFOA4FSSoU5DQRKKRXmNBAopVSY00CglBsi0ktEjouILQj7vl5EPgv0fpVqKNF+BEqBiGQDtxljlgV4v32AXUCUMaY8kPtWKlD0jkAppcKcBgLV4ojIXBHZISKFIrJZRKY6LbtdRLY4LRsmIq8DvYAPHdVB94tIHxExIhIpItNEZLXLMe4RkcWOz5eLyFoROSYiP4vIPKdVVzjejzj2fbaI3CwiK532dY6I/CgiRx3v5zgt+1pEHhaRdEeePxORTo5lsSLyHxE5JCJHHNt2DfgJVS2eBgLVEu0AzgPaAX8C/iMi3UXkGmAecBNwCnAFcMgYcyOwB5hkjGljjHnCZX+LgYEi0t8p7TpgoePzCcc+44HLgTtFZIpj2fmO93jHvr913rGIdAA+Bp4COgJ/Bz4WkY4ux7oF6AJEA3Mc6TMcP2NPx7Z3AMVenB+latBAoFocY8zbxphcY0yFMea/wDZgBHAb8IQx5kdj2W6M2e3F/oqAD4DpAI6AcBpWgMAY87UxZqPjeBuARcAFXmb3cmCbMeZ1Y0y5MWYRsBWY5LTOv40xWcaYYuAtIMWRXoYVAE41xtiNMRnGmGNeHlepKhoIVIsjIjeJyDpHdckR4AygE9aV844G7nYhjkCAdYX+viNAICIjReQrEckXkaNYV+advNxvAuAajHYDiU7f9zt9LgLaOD6/DiwF3hSRXBF5QkSivP6JlHLQQKBaFBHpDbwA3AV0NMbEA5sAAX4G+nnYtL7mc58BnUQkBSsgLHRathDr7qCnMaYd8JzjeN7sNxfo7ZLWC8ipZzuMMWXGmD8ZYwYD5wATsaqolPKJBgLV0rTGKnzzAUTkFqw7AoAXgTkiMlwspzoCB8ABoK+nnTqafr4D/BXoAHzutLgtUGCMKRGREVh3DJXygYo69r0EGCAi1zkeTP8CGAx8VN8PKiJjRSTZ0dfhGFZVkb2+7ZRypYFAtSjGmM3A34BvsQr3ZCDdsext4FGsK/hC4H2sQh3gMeAhR3XSHNxbCFwMvO3SJ2AW8GcRKQT+iFWPX5mfIscx0x37HuWS30NYV/K/BQ4B9wMTjTEHvfhxu2EFp2PAFmA58B8vtlOqBu1QppRSYU7vCJRSKsxpIFBKqTCngUAppcKcBgKllApzkaHOQEN06tTJ9OnTJ9TZUEqpZiUjI+OgMaaza3qzDAR9+vRh9erV9a+olFKqioi4HVJFq4aUUirMaSBQSqkwp4FAKaXCXLN8RuBOWVkZe/fupaSkJNRZadJiY2Pp0aMHUVE6SKVSytJiAsHevXtp27Ytffr0QUTq3yAMGWM4dOgQe/fuJSkpKdTZUUo1EQGpGhKRl0UkT0Q2eVguIvKUiGwXkQ0iMsxp2aUikulYNreheSgpKaFjx44aBOogInTs2FHvmpRP0tal+ZTekG0Cld7Sj13fsoYK1DOCV4BL61h+GdDf8ZoJLABwDJ/7rGP5YGC6iAxuaCY0CNRPz5HytfBZsH6BT+kN2SZQ6S392PUta6iAVA0ZY1aISJ86VpkMvGasoU6/E5F4EekO9AG2G2N2AojIm451NwciX0qFs7R1acxKmVUrfcH6BbXSi8qKWLB+ARP7TqTEXkJJueNlt+4eP9n1CeUV5diN3XqvsKY9+M/m/1BhKjCYGu8AL2x4AYPBGEPlP4AF66yCrPJ75fuz6551+3P4kl45mvIza59xu02w0xvrGIEWsGGoHYHgI2PMGW6WfQTMN8asdHz/AngAKxBcaoy5zZF+IzDSGHOXm33MxLqboFevXsN3767ZL2LLli0MGjQoID9LIN12223ce++9DB7c4Budek2YMIGFCxcSHx9fI33evHm0adOGOXNqDq/fVM+VCqzkV5PZOGMjxhgOFB1gy6EtbC3YStr6NMb0HENBcQGHSg5RUFJAcbnOed8c3TnkTrfB3hMRyTDGpLqmN9bDYnf1EaaO9NqJxjwPPA+Qmprqd/R6f20Of12aSe6RYhLi47hv/ECmDE2sf0MfvfjiiwHfp6slS5YE/Riq+ThpP8nKvSsBuOPzO9hSsIWCkoIa63z989cAdIjp4DYITEiawNUDriYuMo7pH0/ng8kfYIuwERkRiU2s97FvjWXltJVESAQREoEgiAiCcNYbZ7HmhjXWzgRrGULK6ymsv2k94vjTr6yqrAxarnxND+S+muKx61vWUI0VCPZiTRxeqQfWXK3RHtKD6v21OTz4v40Ul1m3tzlHinnwf9aJ9ScYnDhxgmuvvZa9e/dit9v5wx/+wIIFC3jyySdJTU3lpZde4vHHHychIYH+/fsTExPDM888w80330xcXBxbt25l9+7d/Pvf/+bVV1/l22+/ZeTIkbzyyisALFq0iL/85S8YY7j88st5/PHHgeohNzp16sSjjz7Ka6+9Rs+ePencuTPDhw/372SpJs25+ie/KJ/7lt9HRl5G1fL03HQAzk08l1+d+SsGtB/AyIUjfS58+sa7n2mzXUw7j3mLsrlvohwh2n2pqWmsQLAYuMvxDGAkcNQYs09E8oH+IpKENVn3NGrO99ogf/rwJzbnHvO4fO2eI5TaK2qkFZfZuf+dDSz6YY/bbQYnnML/TTq9zuN++umnJCQk8PHHHwNw9OhRFiyw6kNzc3N5+OGHWbNmDW3btuXCCy9kyJAhVdsePnyYL7/8ksWLFzNp0iTS09N58cUXOeuss1i3bh1dunThgQceICMjg/bt2zNu3Djef/99pkyZUrWPjIwM3nzzTdauXUt5eTnDhg3TQNDCLVi/gPN7nM9/tvyHpdlLsVfYGdNjDNMHTedXn/8qIFeOdw6506f0hmwTqPSWfuz6ljVUQAKBiCwCxgCdRGQv8H9AFIAx5jmsCbonANuBIuAWx7JyEbkLWArYgJeNMT8FIk91cQ0C9aV7Kzk5mTlz5vDAAw8wceJEzjvvvKplP/zwAxdccAEdOlhT5F5zzTVkZWVVLZ80aRIiQnJyMl27diU5ORmA008/nezsbHbv3s2YMWPo3NkaOPD6669nxYoVNQLBN998w9SpU2nVqhUAV1xxhV8/j2rathzaAsD0j6fTOqo10wZOY/pp0+l1Sq86t/O18PFUB11X3bSv2wQqvaUfu75lDRWoVkPT61lugF97WLYEK1AETH1X7qPnf0nOkdr1oonxcfz3V2c3+LgDBgwgIyODJUuW8OCDDzJu3LiqZfU9lI+JiQEgIiKi6nPl9/LyciIjvftVafPQli9tXVqtJoQnyk7QNrptjSAQyIJdtWxhWVl33/iBxEXZaqTFRdm4b/xAv/abm5tLq1atuOGGG5gzZw5r1qypWjZixAiWL1/O4cOHKS8v59133/Vp3yNHjmT58uUcPHgQu93OokWLuOCCC2qsc/755/Pee+9RXFxMYWEhH374oV8/j2qakjslE2OLIamd1Tt844yNbJyxsVZBrgV7C1S4H/59GRQeCOhuwzIQTBmayGNXJpMYH4dg3Qk8dmWy362GNm7cyIgRI0hJSeHRRx/loYceqlqWmJjI7373O0aOHMnFF1/M4MGDadfO84M2V927d+exxx5j7NixDBkyhGHDhjF58uQa6wwbNoxf/OIXpKSkcNVVV9WomlItw9Lspdz91d30bdeXVy59JdTZUd7wVHj7mg6w/AnY8x0sfzygWQxYP4LGlJqaalwnpmkObeOPHz9OmzZtKC8vZ+rUqdx6661MnTq10fPRHM6VqiltXRrdW3dn3rfzSOmcwjMXPUPb6LYeO42pICncD+/cAle/Am27erfso3sh498w/BaY+HfP6fZyOHkMPrkfNr4DAy+DYTfByUJ4/06oKK+dn8gYeCjP6+x76keggaARzZkzh2XLllFSUsK4ceP45z//GZI6/eZwrlRNya9ajQdGJ4zmH2P/QVxkXIhz1EJ4Krx9LdSdl505Dc79DTx3LthLg5PvyDgYNBHGPVo7INUh1B3KFPDkk0+GOguqGXpjyxsAXNL7EuafN59oW3SIc9QMeSrYnatanAv2yvQv/wzn3gtpo2oW6qtfsl4SAcZQox/s+oXWy52IKGjVEcpLrKt/UwFigw59oc95kJMB+ZvBXga2GCvt/PugQxJ8+Qis+w/YosF+EmJO8SkI1CUsnxEo1RykrUsj+dVk5v8wH4DPd3/O8P8MD8roky2Gp/p15wLfXgYPd4F57azC3FRY7/PaVb8q09f+B54e5v7KPqoVdBkMfc6F+F4Q4biujoiCnqPgqpdg0BWAWIW6RFhVPXMy4fQrrXUjYwEDSefDpH9A4nCosFvpFWXQvjf0HmUV+MUF1p3Ibcus9+OBe2CsdwRKNVGzUmZRXF7Ma5tfo8JUBHxYgSbP12obgK/nw+5v4aN74PQptevWK6/k3bHFQLueUFoIJw6CsVuFesJQGDETtiyGLR9CZLQVTIZMr76L+PAeWPOKVYDbS6Hr6ZB8Nfz0HqTeCqm3wOp/VxfeJ/KswtzbdIBpb1R/dq2W8pMGAqWaqLyiPBZtXcTEvhNZvGNxqLMTPL5U2xgDn/+fVdi/exv0PgeO7IENixxVNA6ZH1svsArzinLAWNUwnQda9fjZ38D2ZVZVS0UZDL3BOo5rod4tGc68Bja/775QB88FuKfC29f0INNAoFQT9cKGF7BX2LljyB0ktgn8gIhNhmuB/0gXKD9ZvdzTVXz2CusFVpXKiXw4lmsV+rZo6HchXDof0p+yCnZbjFWw9zobzp0Ne3/w7Wq9rkI6RAV4oGggCJFvvvmGO+64g6ioKBYsWMDhw4eZMGGCx/Vzc3O5++67eeeddxoxlypUco7n8M62d5jafyo92/ZsGU1EXa/8vS3wAaLbQNczoOggHN5tXcFHxsJpl8P4x6z9uV7Jn5JoPWQN1NV6CxbeD4uD1EvPG2+88QZz5sxh3bp1ZGZm1juUdEJCggaBMPLc+ueIIIKZZ84MdVZ85+nvqrJKZ9F0+O+N0LpL7W3j2sPgqdYVPmJd2UsEnPkL+OVS6HO+VXdfWdjHxldXJ1UW+K4PU6e9YRXo3ZKtd+eCXgHhfkfgqelYA7kbhrpTp07MmTOH8vJyzjrrLBYsWMDrr7/OW2+9xdKlS/nss89IT0+nuLiYlStX8uCDD9KtWzdmz54NWGMHrVixgkOHDjFx4kQ2bdpEUVERN998M1u3bmXQoEFkZ2fz7LPPkpqaSps2bZg9ezYfffQRcXFxfPDBB3TtGpgmZqpxZB/NZvGOxVx32nV0a90t1Nnx3fInYM+38NFvoOdI+OJPVgucSrkZ1ksiIGEY5K4Fm6Me//Qrrb/FN6/3rdoGwvJKPlBaZiD4ZC7sr6OFxZ70mg+WqtoEC/Qa7X6bbslw2fw6D+tuGOozzjiDL774ggEDBnDTTTexYMECfvOb37By5UomTpzI1VdfzSuvvMLq1at55hlrarpJkybx7LPPMnr0aI4fP05sbGyN46SlpdG+fXs2bNjApk2bSElJqVp24sQJRo0axaOPPsr999/PCy+8UGOoC9X0pa1LI8YWwy+TfxnqrHjmWs1TVABPDrCqbCplLrFeYLXGKdxfXaUzaJLVGerje61WOVptE1LhWTWUcBa06mxdkYD13rozJJ7l126Tk5NZtmwZDzzwAN988w3Z2dkkJSUxYMAAAGbMmMGKFSvq3c/o0aO59957eeqppzhy5EitkUdXrlzJtGnTADjjjDM488wzq5ZFR0czceJEAIYPH052drZfP5NqXJkFmXyS/QnXD7qeTnGdQp0dz9U8Xz5iVfO8fqXVg/aJvlYhL7bqvytbNAyaDL/NglMvqVmlU9kZSqttmoSWeUdQz5U7UPvB0qAr/L7CqGsYal/MnTuXyy+/nCVLljBq1CiWLVtW466grmFBoqKiqoatsNlslJe7GZ9ENVlzls+hbVRbbj795lBnxeJcfTr6bnh6eM12+XmbrPeISJjxEaxfBGtfr/67at3JKvDrqtJRIdcyA4E3gvAfMzc3lw4dOnDDDTfQpk0bnnvuObKzs9m+fTunnnoqr7/+eq2howHatm1LYWFh1fcdO3aQnJxMcnIy3377LVu3bq1R/XPuuefy1ltvMXbsWDZv3szGjWHW0aiF2pi/kexj2fw65dd1TgHZKOpq0RPdFsqLrYDgOubNt8/43vRShVygZii7FPgn1ixjLxpj5rssvw+43umYg4DOxpgCEckGCgE7UO5uQKSgCMJ/zI0bN3LfffcRERFR1Sz06NGjXHPNNVUPi++4445a240dO5b58+eTkpLCgw8+yMqVK/nqq6+w2WwMHjyYyy67jH379lWtP2vWLGbMmMGZZ57J0KFDOfPMM30a0lo1Tc+ufxaAGwff2PgHd67zLz0OZ820BlArPW4tlwjodiZc9gSsf9PpbtplzBst8Jslv0cfFREbkAVcgjVJ/Y/AdGPMZg/rTwLuMcZc6PieDaQaYw56e8zmOvpooNjtdsrKyoiNjWXHjh1cdNFFZGVlER3t3WBk4XSumgN3M46BNcNYo/UfeH8WrFtoVeWcyAcE2nS1rugre95Wjrj55vXWMuerfq3bbxaCOfroCGC7MWan40BvApMBt4EAmA4sCsBxw1ZRURFjx46lrKwMYwwLFizwOgiopmdWyiz6xfdjzvI5AMEdU8i1tc/DnWsOqHYi33q3RUOP1NoFPuhVfwsUiECQCPzs9H0vMNLdiiLSCrgUuMsp2QCfiYgB/mWMed7DtjOBmQC9etU9OXdL17ZtW1zviFTztip3FW2j2lJYVlj/yv6ofPj7wa+tah17qdXSBxytejyMc68FfosWiEDgbmYVT/VNk4B0Y0yBU9poY0yuiHQBPheRrcaYWm0sHQHiebCqhvzNtFJNhTGGlTkrGZUwilPjTw3OQVwf/m7/3HqPiLR67a5f5L7OX4WFQPQj2Av0dPreA8j1sO40XKqFjDG5jvc84D2sqialwsb2I9vJK8pjdMLo4DwTqKiwxuOJalWdZouGM66GezZDydGgjXOvmodA3BH8CPQXkSQgB6uwv851JRFpB1wA3OCU1hqIMMYUOj6PA/4cgDwp1Wyk56QDMDrRQ6/2hqh8FjDyTlj5D8hdY43jU1ZszXNrL4XYdtWduippFVBY8jsQGGPKReQuYClW89GXjTE/icgdjuXPOVadCnxmjDnhtHlX4D1HB6hIYKEx5lN/86RUc5Kem86p8acGdlyhTx+E3aus1ymJMGUBbP3Y/cNfFfYC0o/AGLMEWOKS9pzL91eAV1zSdgJDApGHUDty5AgLFy5k1qyG3dqPGTOGJ598ktTUxulGoZqGorIiMg5kcN1ptW6iG8b1WQDAsRxrALiH8qrT9MpfOQnPsYacBGr+1yNHjpCWpnPJKt+sPrCasooyzkk8x/+dHdoBnQZanytbAkXGQfI1MFt7nyvPwj4QuOvI0xBz585lx44dpKSkcM8993DRRRcxbNgwkpOT+eCDDwDIzs5m0KBB3H777Zx++umMGzeO4uLiqn28/fbbjBgxggEDBvDNN98EJF+qaUvPSSfWFsvwrsN937hqQLj9sOY1eO48a9rGvmMBo62AlNda5FhDj//wOFsLtnq9/i2f3lLvOqd1OI0HRjzgcfn8+fPZtGkT69ato7y8nKKiIk455RQOHjzIqFGjuOKKKwDYtm0bixYt4oUXXuDaa6/l3Xff5YYbrOfn5eXl/PDDDyxZsoQ//elPLFu2zOufQTVP6bnpnNXtLGJsMb5vvPwJawTQl8bBkd2QdD5MeQ4+uV8HeFM+aZGBoD45x3PYd6J67J7VB6zOWd1bdw/I3LDGGH73u9+xYsUKIiIiyMnJ4cAB648xKSmpagA512Gir7zySrfpqmX6+djP7D62m+mnTfdtQ9fnAEd2O3b4PbRL1FZAymctMhDUdeXuKvnV5IB36X/jjTfIz88nIyODqKgo+vTpQ0lJCQAxMdVXfjabrUbVUOUyHT46PKTnOpqNJvjYbHT2Bnj3Nsh2VB/aYmDwFVZvYKUaIOyfEQSK81DSR48epUuXLkRFRfHVV1+xe/fuEOdONUXpuekktkmk9ym9fdvw5+8he6X12RZjDQinzwGUH1rkHYEv7hxyZ0D207FjR0aPHs0ZZ5zBWWedxdatW0lNTSUlJYXTTjstIMdQLUeZvYzv933PFf2uqJpIyCvrFlrjBMXGW2MCjfyVPgdQfvN7GOpQCPdhqP2l5yr0ftj3A7/87Jc8NfYpxvYaW/fKlb2E+10EXz4MfcfAL96AmDaNklfVcgRzGGqllI/Sc9OJlEhGdPdiaK3lj1utg3avgoGXw9UvQ1Rs/dsp5SUNBEqFQHpOOkO7DqV1VGvPK7nrJZz5MTzeq2YvYaX81KIeFjfHaq7Gpuco9PKK8sg8nFl/a6HZG6DPudXftZewCpIWEwhiY2M5dOiQFnR1MMZw6NAhYmO1WiGUVuWuAuDcxHPrXrGiHH7+wfocGaO9hFXQtJiqoR49erB3717y8/NDnZUmLTY2lh49eoQ6G2FtVY4VCAa0H+B5JXsZvHMrVNiteQPO/Y22DlJB02ICQVRUFElJSaHOhlJ1slfYWbXPCgR1Nhv94k9Wf4GrXoLkq6007SWsgqTFVA0p1dSlrUsj5fUUjp48Cli92pNfTa49Am7mJ7DqaUj9ZXUQUCqIWswdgVJN3ayUWSS1S+L+FfcD1B7apHA/vHkdHNwG3c6E8X8JQS5VOApIIBCRS4F/Ys1Q9qIxZr7L8jHAB8AuR9L/jDF/9mZbpVqSzIJMIiMiKa9wM5bU149BTgZERMG1r4ZFX4H31+bw16WZ5B4pJiE+jvvGD2TK0ESf0wO5r6Z47LqOEQh+9ywWERuQBVyCNZH9j8B0Y8xmp3XGAHOMMRN93dYddz2LlWoO7lx2J3lFeVzU66Lqierd9RcAq6VQM+sv4EsBB/Dg/zZSXGav2j4uysZVwxN5NyPH6/THrkxu4L72UlxWUZUeGxXB1KGJvLc2hxKX9MlDEvhgfW6t9D9OGgwG/vzR5lrLJp2ZwIcbam8z8cwEPnKb3p2PNuyrlf7Q5dYoAI98vKXGssqf3Zdg4KlncSACwdnAPGPMeMf3BwGMMY85rTMG94Gg3m3d0UCgmquL3rqIkd1H8pfznKp9CvfDh7MhyzFdd2ScNY7QuEebZFPRugp7d4XxlKEJvLcmh5Ly6kIs2iZE2iIoKrXX2r8AvpRKtggBY7CHYcvxxPg40ude6PX6wRxiIhH42en7XmCkm/XOFpH1QC5WUPjJh20RkZnATIBevXoFINtKNa7DJYfJK86r3Wy0bTfruQBYo4k2kf4C3lzF5xwp5v531rMiK4/PNufVCAIAxWV2Fv3wc619l9oNpfbaQQB8CwIA9oowjAAOuUeK61/JC4EIBO7awLn+ZtYAvY0xx0VkAvA+0N/Lba1EY54HngfrjqDBuVUqRLIOZwEwoINLIDi4DQp2QNdkmLqg0fsLeFvgz3l7PSJQ5nLpXWo3/G9tbsDyYxPB7qamwlN6YnxcVR793VdTPHZdx0hwLPNXIJqP7gV6On3vgXXVX8UYc8wYc9zxeQkQJSKdvNlWqZYisyATgIHtB9ZcsOKvVnXQje9Bt2Srv4DzLGNBVFmdk3OkGEPlFf4G5r67odbVfXmFqRUEKgmQEO/+4bbNQ3+J+Lgo4qJsNdLiomxMH9nTp/T7xg/kvvEDA7Kvpnjsuo5RGbT9FYg7gh+B/iKSBOQA04DrnFcQkW7AAWOMEZERWAHoEHCkvm2VaikyD2fSKa4THeM6Vice3AYb34azfw1tOgf1+O6u/J9YurVWgV9qr/CwB88q9+fLA9t5V5wO4PZ5Q2rvDj6lVwrEvprises6RiAEZD4CR3XP/8NqAvqyMeZREbkDwBjznIjcBdwJlAPFwL3GmFWetq3vePqwWDVH1354Le1j2/OvS/5Vnfju7bD1I2uAuSAGAncPcm0R4nP9enxcFCfLK9y23Glos0jVeILWaigUNBCo5qasooyRb4zkhkE3cG/qvVZifhakjYSz74JxDwf1+KPnf+m2jtmTugp8CN6VqQounZhGqRDKPppNWUVZzQfFK56AyFgYPTugx3K9+r75nD51BoG4KJtP1TaAFvwtjAYCpRpBVYuhyqaj+Vmw8R0YfTe07hSw47hWAeUcKebRJVs8rp/oKOC1wA9vGgiUagSZhzOJiogiqZ1jhNzlj0NUKzjn7oAe569LM2s9/AU4JTaSMrupdeVfWehrgR/edPRRpRpBVkEW/eL7ERURBdnpsOkdSLkuoHcD9grjsQqosKScx65MJjE+DsG6E/B1eALVcukdgVKNIPNwJucknGN9WXyX9V5W1OD9uT4HuOmc3nz2k+dOaAnxcXrlrzzSOwKlguxQ8SEOFh9kwOo3YF47KNhpLVjn+P5IF5/2564T2GNLtrI59yjXj+pJXFTNP+tAdjxSLZMGAqWCrPJB8cBJaZAwrHpBAyej9/QcoF1cNI9OOZPHrjxTq4CUT7RqSKkgq2oxlDgKjjuGlfZjMnpPA40dOFYCoFVAymd6R6BUkGUWZNI5rjMdImKhMBe6nA63fQHDb2nQ4HIdWke7TQ/UAGQq/OgdgVJBlnU4y+pItms5mAoY/0j14HI+euvHnyk4UYpYQ/BX0ecAyh8aCJQKojJ7GTuO7mB04mhr4pnoNtD7XK+3d24d1CY2ksKScs7r34nLk7vz9JfbdZgHFRAaCJQKop1Hd1JeUc7A9gPg639Bvwsh0n3VjivXXsKFJeXYRJiSksBVw3sybYRO0KQCQ58RKBVEVQ+Kyw0U7oOBl3m9rbvWQXZj+Pvn2wKaR6U0ECgVRFmHs4iOiKZPzgZA4NRLvN7WU+ugQE1PqFQlDQRKBVFmQSb94vsRue0z6HGWT3MOnBLnvuZWWwepQAtIIBCRS0UkU0S2i8hcN8uvF5ENjtcqERnitCxbRDaKyDoR0UkGVIuSeTiTAW16Qu5aGDDe6+0+3bSfo8XlRLjM8qitg1Qw+B0IRMQGPAtcBgwGpovIYJfVdgEXGGPOBB7GMQm9k7HGmBR3EyYo1VwdLD5IQUkBA0vLrAQvnw9k7D7M7DfXMrRXPPO1l7BqBIFoNTQC2G6M2QkgIm8Ck4HNlStUTkvp8B3WJPVKtWhZBY6hJQ7ugnY9oYvr9VE152aiItChVRQv3pRKxzYxXHtWz8bKsgpTgagaSgR+dvq+15HmyS+BT5y+G+AzEckQkZmeNhKRmSKyWkRW5+fn+5VhpRpD5uFMAAbsyYABl4KI2/VcB5GrMFB40s432w42Ym5VOAtEIHD3v9vtRMgiMhYrEDzglDzaGDMMq2rp1yJyvrttjTHPG2NSjTGpnTsHb5JvpQIl83AmXaLbEX/yhBUIPHDXTPRkeQV/XZoZ7CwqBQQmEOwFnO9dewC5riuJyJnAi8BkY8yhynRjTK7jPQ94D6uqSalm79vcbxlooiCqNfTx3JtYm4mqUAtEIPgR6C8iSSISDUwDFjuvICK9gP8BNxpjspzSW4tI28rPwDhgUwDypFRIldpLrQfFR/Oh31iIivW4bsc2OoicCi2/HxYbY8pF5C5gKWADXjbG/CQidziWPwf8EegIpIlVT1ruaCHUFXjPkRYJLDTGfOpvnpQKtZ1HrclnBhw/BCM9VwsVnCjlZJkdoWZ9qjYTVY1JjHFbnd+kpaammtWrtcuBanrS1qWxYP2CWul3DrmTWSmzaqQZY7jt1dV8s+0gsy/uz8Lv9+ggciqoRCTDXTN9DQRKBcHTb13B88W7yMgrIvq+HW7XeSV9F/M+3My8SYO5eXRSI+dQhSNPgUBHH1UqkB7pAuUn2dWlE7RuRfSJg9a8xJEx8FBejf4CBhjcvS0zzukT6lyrMKdjDSkVSLM3wBnXsCsqit6lZWCLqZqX2LW/AMDOgyf4YF2tRnZKNSoNBEoFUttu2KPbsDsqkguLiqGirGpeYnf9BUrKtL+ACj2tGlIqwHJP7KVMhKTo9jD8qqp5ibW/gGqq9I5AqQDbdd7dACR1GmzNSzztDQC6tXPfl0D7C6hQ00CgVIDtytsIQJ9uw2qkn9q5da11tb+Aago0ECgVYLvy1tPBbic+sbqVXsbuAlbuOMR5/TvpsNKqydFnBEoF2K6jO+lTVgZdkwE4WW7ngXc3ktAujuduGE7rGP2zU02L3hEoFWC7Sg6RRDS07gjAs19uZ3vecR6deoYGAdUkaSBQKoAOlxzmMOUkteoGwNb9x0j7egdThyYyZmCXEOdOKff08kSpAMp2zEq2c387kuZ+TKRNiImM4A8TPc9OplSo6R2BUgH0ScZnAOw53gcDlNkNpfYKVmTprHqq6dJAoFQArc3OILrCsKv0tKq0MrvR3sOqSdNAoFQAlUYcoGd5OXtM9xrp2ntYNWUaCJQKoBMxJ+hQGk2Fy5+W9h5WTVlAAoGIXCoimSKyXUTmulkuIvKUY/kGERnm7bZKNRel5Sc5GFlB5Mn2NdK197Bq6vwOBCJiA54FLgMGA9NFxLWJxGVAf8drJrDAh22VahZ+zv0RuwgnS7vTuU2M9h5WzUYgmo+OALYbY3YCiMibwGRgs9M6k4HXjDUd2nciEi8i3YE+XmyrVLPww5YvAejXYwSv3nBxiHOjlPcCUTWUCPzs9H2vI82bdbzZFgARmSkiq0VkdX6+NsVTTYu9wpC+7QcAbr9sWohzo5RvAhEIxE2a60TIntbxZlsr0ZjnjTGpxpjUzp07+5hFpYJr4fe7KTQH6GKH7h0TQp0dpXwSiKqhvUBPp+89ANe59zytE+3Ftko1Sc7zDwMM61tE36h2Ic6VUr4LxB3Bj0B/EUkSkWhgGrDYZZ3FwE2O1kOjgKPGmH1ebqtUk+M6/3BrTpATBe1E71ZV8+P3HYExplxE7gKWAjbgZWPMTyJyh2P5c8ASYAKwHSgCbqlrW3/zpFSwuc4/fGpUJjsiIthzsFMIc6VUwwRk0DljzBKswt457Tmnzwb4tbfbKtXUufYU7haTxQ5g17Gk0GRIKT9oz2KlGsC1p3CbmL0ARMYMCkV2lPKLBgKlGuAXZ/Wo8b0i+hAxFXD/JSNDlCOlGk4DgVI+Ki2vYMnG/ZwSa6N7u1gisXMspoSeEW2ZOqxH/TtQqonRiWmU8tHzK3awdX8hL9yUyiWDu0J+JpcstpHavneos6ZUg+gdgVI+2J53nKe+2M7lyd2tIAAU5WSwPzKSpI46TJZqnjQQKOWligrDg//bQFy0jXlXnF6Vnp3zHQBJ3VNDlTWl/KJVQ0rVo7IHcY6jyej0ET3p3DamavmuQ9YYiUnt+4ckf0r5S+8IlKqDcw9i57T31+ZUfd91PIcIoNcpvUKQQ6X8p4FAqTq49iAGKC6rqJ6DeP8mdpmT9IhsS7QtOgQ5VMp/GgiUqoOnuYar0r/4M7uio0iyux00V6lmQQOBUnXo5PQswNnW2Bkwrx32bUvZHRlF0uFcmNcOHunSyDlUyn8aCJTy4MTJcjC1r/Tjomx8OX4ZnHENuVFRlEYISRUCydfA7I0hyKlS/tFAoJQH8xb/xMETpfx6bD8S4+NqzEF82dkpEBnDrkgbAH1OFkPMKdC2a0jzrFRDaPNRpdxYvD6XtzP2ctfYU5kzfiD3jT+t9kp5m9kVFQVA0uBr4PiBRs6lUoGhgUApB9cZx3p3iGP2xe77BqStS2NBzEGIaQ/A+YeXA3DnujRmpcxqnAwrFSBaNaQUtWccM8D+Yyf5eMM+t+vPSp7Jxv2FDIpoDcDGGRvZOGOjBgHVLPkVCESkg4h8LiLbHO/t3azTU0S+EpEtIvKTiMx2WjZPRHJEZJ3jNcGf/CjVUO76C5wsd+ov4Grvj5QXH2aHOdkIuVMquPy9I5gLfGGM6Q984fjuqhz4rTFmEDAK+LWIOI/O9Q9jTIrjpTOVqZCot7+Aq6xP2B0TS6kp58KeFwYxZ0oFn7+BYDLwquPzq8AU1xWMMfuMMWscnwuBLUCin8dVKqA6tnHfK9h1JrIqWUvJTLAGntPqINXc+RsIuhpj9oFV4AN19qYRkT7AUOB7p+S7RGSDiLzsrmrJaduZIrJaRFbn5+f7mW2lqmUdKKSwpAxxSY+LsnHf+IG1NyjYBflbyezQg8iISPq269so+VQqWOoNBCKyTEQ2uXlN9uVAItIGeBf4jTHmmCN5AdAPSAH2AX/ztL0x5nljTKoxJrVz586+HFopjw4cK+GWf//IKXHR/GHioFr9BaYMdXPzmrXUeouMoG+7vkTZoho300oFWL3NR40xF3taJiIHRKS7MWafiHQH8jysF4UVBN4wxvzPad8HnNZ5AfjIl8wr1RDOzUQjbdZ9wHuzRnNGYjtuPdeLq/usT6DTQLIKf2Zkd52jWDV//lYNLQZmOD7PAD5wXUFEBHgJ2GKM+bvLsu5OX6cCm/zMj1J1cm0mWmY3CML2vOPe7aDkGGSnc/jUseQV5zGwg5uqI6WaGX8DwXzgEhHZBlzi+I6IJIhIZQug0cCNwIVumok+ISIbRWQDMBa4x8/8KFUnd81ES+11NBN1teNLqCgjs9sAAPrrZDSqBfCrZ7Ex5hBwkZv0XGCC4/NKqPUcrnK9G/05vlK+8rmZqKuspRDXnizHGEMD2+sdgWr+tGexChvHT5YTHen+v7zHZqLOKuywbSmcegmZR7bRKa4THeM6BjiXSjU+DQQqLOQVljDt+W8pLa8gylbzBtVjM1FXe1dD0SEYeClZh7P0bkC1GDronGqRnFsGdW4bQ5m9gpKyCl6++SyOFpdVLUuIj+O+8QPdNxN1lfUpRERSlnQBO9Y+ytmDzg7+D6JUI9BAoFqcypZBlQ+F8wqt8YDuvaQ/Y0+z+jx6VfC7yvoUep1NdulhyirKGNBhQMDyrFQoadWQanHctQwC+O+Pexu+072rIW8z9B5N5mGrhZFWDamWQu8IVIvjd8sgd5Y+ZL0f2kFW2xiiIqLo065Pw/enVBOigUA1a87PArrHx3JOv04e1/WqZZCrR7pAudNQ05veJiu/M/0iI4mK0KElVMugVUOq2XLtJZx7pIR3MvbStW0MMS7NRL1uGeRq9gZIuqD6e2Qcma1PYcCpl/uXeaWaEA0Eqtny9CzAZhMev+pM7waQq0+brpC/1focGcMhU8ZBU8bALkP8y7xSTYhWDalmwbkKKCE+jjvG9CXH47OAEqYMTWxYwe9q1wprUvpeZ8OEv5K16kkoXKMthlSLoncEqslzrQLKOVLMH97/yeP6DXoW4I4x8PV8aJsAN74P3ZLJGmjNRqYthlRLoncEqklxvfK/b/xAnvh0q9sqoLYxNsorqLGswc8C3Nm1HPasgglPQlQsAJkFmXSJ60L7WI9zKCnV7OgdgWoy3F35//at9eQeLXG7/vGTdh67MjkwzwJcOd8NDK0eGzHzcKZWC6kWR+8IVNC4u7qvLKRdl825ZACPubnytxuDAMbN/hPi4wL3LMDVzq9hz7c17gbK7GXsPLqTcxPPDfzxlAohDQTKb+4KfKDGMA85R4p58H8bATDG8OB7Gykpq6hads/b6z3u32BV+QStCqjWAZ3uBobdVJW88+hOyivK9fmAanH8CgQi0gH4L9AHyAauNcYcdrNeNlAI2IFyY0yqL9sHgqerU1+uWuvbJlDpzenY4L7Aj42KqHV1X1xm5/53NlBWUYFxc4kvgtv0RMexGjRQXEPs/Bp+/s66G4iMqUrOOpwFoLOSqRZHjLu/PG83FnkCKDDGzBeRuUB7Y8wDbtbLBlKNMQcbsr2r1NRUs3r1aq/z6ToIGVhXlFcNT+TdjJxa6Y9dmQzg0zaBSm+ax45gytBE3lubU3UVDxBlE6JsERSV1n6Q21DurvwDVu/vjWP74NmzIKo1/GZDjUDwt9V/Y+GWhXx//fdERujNtGp+RCSj8kK8RrqfgSATGOM0ef3Xxphal0t1BAKvtnflayAYPf9Lt23OIwQq3Pz4ndpEA3DweKnX2wQqvUPrKDBQUFTm9TaerqQ91a17So+OjMAYQ5m94f8n6pPoaNrp7vfR6Ff+7iz8hWOU0XPg1k9qLJr52UyOnDzCW5Pearz8KBVAngKBv5c1XY0x+wAchXkXD+sZ4DMRMcC/jDHP+7g9IjITmAnQq1cvnzLpabAxd4UquA8A9W0TqPSCE7UDQH3beIrlnopzT+ml5RUelvguPi6Kk+UVHuv13d3xVBb6jVrwV3IdU2jPKpjXzrojeCgPsFoMnd/j/MbPm1JBVm8gEJFlQDc3i37vw3FGG2NyHQX95yKy1RizwoftcQSP58G6I/Bl24T4uGZzR9ClrVUVUTmGvj/7sgm4u7i3iWB3Ez3qulr3tI2nAn/eFacD1Hl1H9Irf1ezN8DrUyBvi/U9Mg4GTYRxjwJwsPggBSUF+qBYtUj1BgJjzMWelonIARHp7lS1k+dhH7mO9zwReQ8YAawAvNreX/eNH+hTnftDlw8GQlNP/7sJg0J27Lqu1j1tU1+B76lwD9mVvyd5m52CQCzYT0LMKdC2KwBZBdaD4gHttQ+Bann8rRpaDMwA5jveP3BdQURaAxHGmELH53HAn73dPhAqCxx3hVVq7w4+X7V62iZQ6c3t2M7nuFk6tg/evR2i28AZV8GI22H1v60xhhyqJqPRFkOqBfL3YXFH4C2gF7AHuMYYUyAiCcCLxpgJItIXeM+xSSSw0BjzaF3b13dcXx8WK+WRvRxenQT71sPMr6Cz+4J+7jdz+Xjnx2ycsbGRM6hU4ATlYbEx5hBwkZv0XGCC4/NOwO2YvZ62V6rRfPWo9WB46vNug0DaujQWrF9Q9T35VauJ7Z1D7mRWyqxGy6ZSwaSNoVV4KtwPr0+1ng0MmwFDfuF2tVkpsxjfZzxTPpgCoHcEqkXSQedUePr8j1YQiOsAlz1e56orc1Y2UqaUCg29I1DhxbW/QHEBPNqtRn8BV+k56fRr149xfcY1UiaValx6R6DCy52roFXn6u+RcZB8Dcx2X+VTXF5MxoEMzkk8R58JqBZL7whU+CgqgP/9CoryAbHuAlz6C7havX81pRWlnJugQ0+rlksDgQoPhQesh8OHtkFiKnQfAqm31Oov4Co9N51YWyzDuw1vxMwq1bg0EKiWq3A/vHMLjPsLvPtLKNwH1/0X+l1Yvc7Ev9e5i/ScdFK7pRJji6lzPaWaM31GoFqu5U/A7m/hlcvhxEFrAnrnIFCPvYV7yT6WzeiE0cHLo1JNgN4RqJbHtWVQ2Qnr/bVJHlsGubMqdxUAoxM1EKiWTe8IVMtzVwZ0HlT9PTKmzpZBnqTnpJPYJpE+p/QJbP6UamI0EKjmr3A//Psy64HwoR3w9gzId4wkaosBe1mdLYPcKbOX8f3+7zkn4RxEJEgZV6pp0Koh1fwtfwL2fGc9GM5dB7YoSBwG3Yd61TLInXX56zhRdkKrhVRY0ECgmi/XZwG70633ihi4/avq9HpaBrmTnpNOpEQysttIPzOpVNOnVUOq+Zr1PXQ9o/p7RJT1LOA3/g8Mtyp3FUO6DKFNdBu/96VUU6eBQDUflc8CjuXCmtetzwc2Wcts0WDsPj8LcOdg8UG2FGzh3ETtTazCg1YNqeZj+eNWv4C0c6DksNVDuH0SdBnU4GcB7lQ1G9X+AypM+BUIRKQD8F+gD5ANXGuMOeyyzkDHOpX6An80xvw/EZkH3A7kO5b9zhizxJ88qRbI9VlAieO/2IGNNfsFNOBZgDvpOel0jO2o01KqsOFv1dBc4AtjTH/gC8f3GowxmcaYFGNMCjAcKKJ66kqAf1Qu1yCggOoqoIPb4ds0aNXJscDRjLOeEUP9Ya+wsyp3FecknEOEaM2pCg/+Vg1NBsY4Pr8KfA08UMf6FwE7jDG7/TyuasmWPgS7V0HaKKgog15nQ6eBsOtrR7+AukcM9ceWgi0cOXlEm42qsOLvJU9XY8w+AMd7l3rWnwYsckm7S0Q2iMjLItLe04YiMlNEVovI6vz8fE+rqebEuSNYUQH8uSPMaweb3raWV5RZ77lrILoVDL8FbltmvQfgWYA7lbORnZ1wdlD2r1RTJMaYulcQWQZ0c7Po98Crxph4p3UPG2PcFuYiEg3kAqcbYw440roCBwEDPAx0N8bcWl+mU1NTzerVq+tbTTV1i++GNa9Bux7WyKAV5daVflmR9TkyDgZNhHGPBuXq352bPrmJtXlrdW5i1SKJSIYxJtU1vd6qIWPMxXXs9ICIdDfG7BOR7kBdI3pdBqypDAKOfVd9FpEXgI/qy49qZiqHgr76FaswP54Pfz/NKugrHf3ZerdFwxlXwZpXITI2qFVArtLWpbFg/YKq78mvJgNw55A7dWYy1eL5+4xgMTADmO94/6COdafjUi1UGUQcX6cCm/zMjwoV1wK/UmWTz0XTISIC9q4GjKOgL7Pa/jtf+X98r1X1E8DmoN6YlTKLTQc3sT5/PcdKj+kdgQor/gaC+cBbIvJLYA9wDYCIJAAvGmMmOL63Ai4BfuWy/RMikoJVNZTtZrlqLirH+1k+H0beCQvOrnnVn5thvUfY4LYvIeMV91f+096o3iZAzUG9sS5vHd/kfMPsYbP555p/NtpxlWoK/AoExphDWC2BXNNzgQlO34uAjm7Wu9Gf46sQcL3yd23jv/pl6wXWlb691HHVHwuDJlXX96/ID8mVvydPr32aDrEduO606yi1l4Y0L0o1Nu1ZrNxzV9VjDHz2kFXV8/qVENPaupdzJjZISIHxj8H6RU5X/aU16/tDdOXvzvf7vueH/T/wwFkP0CqqlT4TUGFHA0E481SvD/D1Y1aB/+4voevp8MO/rEBQKc/xOEds0Hcs7PwaIqOtev/uKdBrJKx6qkld9btjjOGptU/RtVVXrhl4Taizo1RIaCBoSTwV7B4f5D4Be76FT+6D0yZC/lZY+Q8wFdXrZH9jvQA69IMje6z2/a4PeFNvrV3gN6Grfk++yfmGDfkb+L+z/08nqFdhSwNBc1Rnwf6d1VLHueD96i/W1f0HsyDpAlg2z6q3r7T5A+sFVmFfXmIV5hXlNev2v54Pa15pMg94/VVhKnh67dP0bNuTyadODnV2lAoZDQShVlf1jLcF/sNdrIK50uqXrJer7cusF0BcByg5agUEW7RVvXP53yC+J3x4j1OB71S3fyKvyVf1+OLz3Z+ztWArfzn3L0RFRIU6O0qFjI6qFWjOwyZ4k+5cqLuqbIP/6QOw/YvqIRhWv2RV36x+yfruHAQqRbWypmqM7wMRjnhvi4HBk+G3WTB4ClXt+SvKrd698T2t9SoLfNfhHKa9YQWebsnWu/OdQDNjr7DzyHeP0K9dPyYkTah/A6VaML0jqE+D6t3dVM84t7O/6I/w5ADrartS5VW8RDgeyjo9mP3pPevlSiKgfR8YNBlyMqy6fFuUVbAPmW4d3/XqvlWn+q/um2lVjy8+3vUxR04e4Y9n/xFbhC3U2VEqpOoda6gpavBYQ74W6gAf3QsZ/7YKTedCsTJ9yHVw3r3WSJn+tj+3RVtX8K07wrEcayauinIrvfdoGPt7WPsarP2PlWYvrc7Xm9dDm641C/Zpb3hOD2NFZUVctfgq9h7fy4abNiAioc6SUo3C01hD4RUIXAv1CjucLIQl98HGt2HgBBh2k1V3/sGsmj1jG8IWZRXCZcVQfNiqzomIhC6nw9AbrPr6bZ9ZhXpFWc1gU3kl722Br7zyj4x/8PKml2ul65hCKhw0eNC5FqFW71cPD1MzP7Zebon1gLVTf+tqvTDXCiQRUZAwFM66DbYshq0fVxfsQ29yXz3TIxVGzoRdy903uwTPVTdhUG0TLIeKD7EqdxWREZE8ft7j/Hb5b3VMIaUIl0Awe4M12cnm960CWmzQsR/0PteqW8/fbHWEskVDn/Pg/Put5V8+DGtfr74qP32K+4K9WzIM+YUVCNwV7A0p1LXAD6j9J/Zz+2e3s//Efp6+8GlrYvrloc6VUk1DeASCtt0gpm31mDf2UqvAryzUD2ysTm/fB3qPsrYrOuS+APe1YNdCPWTS1qVxed/Luf2z2yksLeRfl/yLYV2HAVZ1kFIqnJ4R6MPUsJT8ajIdYztSYSp47pLnGNxxcKizpFTIhPczAtCr9TBTVFbEhzs+BMAWYePlS16mb3zfEOdKqaZJO5SpFiFtXRoA+47v48YlNzJy4Uge+f4RAPKK8pj8weSqdZRSNYXPHYFqVtLWpbltzuku3RjDgvUL2H5kO1/s+QJBGNd7HDcMvoGbPrlJWwYpVQ+/7ghE5BoR+UlEKkSkVr2T03qXikimiGwXkblO6R1E5HMR2eZ4dzvxfSB5uiqs62rR120Cld7Sj13XMuf5g13Tc47nsGz3Mp5a8xR3LruTsW+NBax5BW4+/WY+ufIT/jbmbwztMtTjcZVS1fytGtoEXAms8LSCiNiAZ7Emrx8MTBeRyid2c4EvjDH9gS8c34OqrgImUNsEKr0lH7vCVFBqL2XB+gXkFeWx59geMgsyWZ+/nu/2fQfAG1ve4Om1TzNv1Tzu/vJurl9yPQCXvnsp93x9Dy9ufJGVOSs5VHIIgGOlx3h508u8t716OA5tGaRU/QLSakhEvgbmGGNqNeURkbOBecaY8Y7vDwIYYx4TkUxgjDFmn4h0B742xgys73gN7Vn8r/X/4pl1z9CvXb9ay3Yc3eE2va5lwU73tMxg2Hl0J33b1X742ZD0pHZJtdKNMWQfy6bPKX2qjlmZvqdwDz3b9sQYU5UOkHM8h26tu1FhKjDGWO8YCkoKaBPVBruxU15RTnlFeY3t6mMTG3bnYbMdbk++nbuH3Q1YrYO0CkipugV1iIl6AsHVwKXGmNsc328ERhpj7hKRI8aYeKd1Dxtj3FYPichMYCZAr169hu/evdvr/KWtS3N7VRofE8+Rk0dqpVcWmDuP7vR6m0Clh/LYlQFnx9EdtZa1j2nP4ZOHa6V3iOlAwcmCWumdYjtxsORgrfShnYciIqzJW1Nr2bje45hy6hRiI2O5demtfH3t18THxNcYFM5Tga+BQKn6eQoE1lVdHS9gGVYVkOtrstM6XwOpHra/BnjR6fuNwNOOz0dc1j1cX36MMQwfPtw01BmvnOFTekO2CVR6Sz92IPf17NpnPR5DKWUBVhs3ZWq9rYaMMRf7F4PYC/R0+t4DyHV8PiAi3U111VCen8dSLYSnun1P6TpgnFIN1xj9CH4E+otIkohEA9OAxY5li4EZjs8zgA+CnRlfC5iGbBOo9JZ+7LqWeSrYtcBXKvD8ekYgIlOBp4HOwBFgnTFmvIgkYFUHTXCsNwH4f4ANeNkY86gjvSPwFtAL2ANcY4ypXeHsosHDUCulVBjT+QiUUirMeQoEOsSEUkqFOQ0ESikV5jQQKKVUmNNAoJRSYa5ZPiwWkXzA+67FNXUCand5DT3Nl280X77RfPmmqeYL/Mtbb2NMZ9fEZhkI/CEiq909NQ81zZdvNF++0Xz5pqnmC4KTN60aUkqpMKeBQCmlwlw4BoLnQ50BDzRfvtF8+Ubz5Zummi8IQt7C7hmBUkqpmsLxjkAppZQTDQRKKRXmWmQgEJFrROQnEakQkVSXZQ+KyHYRyRSR8R627yAin4vINse721nT/Mzjf0VkneOVLSLrPKyXLSIbHesFfaQ9EZknIjlOeZvgYb1LHedwu4gEfa5pEfmriGwVkQ0i8p6IxHtYr1HOV30/v1iecizfICLDgpUXp2P2FJGvRGSL4///bDfrjBGRo06/3z8GO1+O49b5ewnR+RrodB7WicgxEfmNyzqNcr5E5GURyRORTU5pXpVDAflbdDdbTXN/AYOAgbjMnAYMBtYDMUASsAOwudn+CWCu4/Nc4PEg5/dvwB89LMsGOjXiuZuHNe1oXevYHOeuLxDtOKeDg5yvcUCk4/Pjnn4njXG+vPn5gQnAJ4AAo4DvG+F31x0Y5vjcFshyk68xwEeN9f/J299LKM6Xm9/pfqwOV41+voDzgWHAJqe0esuhQP0ttsg7AmPMFmNMpptFk4E3jTEnjTG7gO3ACA/rver4/CowJSgZxboSAq4FFgXrGEEwAthujNlpjCkF3sQ6Z0FjjPnMGFPu+Pod1kx3oeLNzz8ZeM1YvgPiHbPwBY0xZp8xZo3jcyGwBUgM5jEDqNHPl4uLgB3GmIaOWOAXY8wKwHUuFm/KoYD8LbbIQFCHROBnp+97cf+H0tUYsw+sPy6gSxDzdB5wwBizzcNyA3wmIhkiMjOI+XB2l+P2/GUPt6PensdguRXr6tGdxjhf3vz8IT1HItIHGAp872bx2SKyXkQ+EZHTGylL9f1eQv1/ahqeL8ZCcb7Au3IoIOet3jmLmyoRWQZ0c7Po98YYT1Neipu0oLWf9TKP06n7bmC0MSZXRLoAn4vIVsfVQ1DyBSwAHsY6Lw9jVVvd6roLN9v6fR69OV8i8nugHHjDw24Cfr7cZdVNmuvP36j/12ocWKQN8C7wG2PMMZfFa7CqP447nv+8D/RvhGzV93sJ5fmKBq4AHnSzOFTny1sBOW/NNhAYYy5uwGZ7gZ5O33sAuW7WOyAi3Y0x+xy3p3nByKOIRAJXAsPr2Eeu4z1PRN7DuhX0q2Dz9tyJyAvAR24WeXseA5ovEZkBTAQuMo4KUjf7CPj5csObnz8o56g+IhKFFQTeMMb8z3W5c2AwxiwRkTQR6WSMCeoAa178XkJyvhwuA9YYYw64LgjV+XLwphwKyHkLt6qhxcA0EYkRkSSsyP6Dh/VmOD7PADzdYfjrYmCrMWavu4Ui0lpE2lZ+xnpgusnduoHiUi871cPxfgT6i0iS42pqGtY5C2a+LgUeAK4wxhR5WKexzpc3P/9i4CZHa5hRwNHK2/xgcTxvegnYYoz5u4d1ujnWQ0RGYJUBh4KcL29+L41+vpx4vCsPxfly4k05FJi/xWA/DQ/FC6sA2wucBA4AS52W/R7rKXsmcJlT+os4WhgBHYEvgG2O9w5ByucrwB0uaQnAEsfnvlitANYDP2FVkQT73L0ObAQ2OP5DdXfNl+P7BKxWKTsaKV/bsepC1zlez4XyfLn7+YE7Kn+fWLfszzqWb8Sp9VoQ83QuVrXABqfzNMElX3c5zs16rIfu5zRCvtz+XkJ9vhzHbYVVsLdzSmv084UViPYBZY6y65eeyqFg/C3qEBNKKRXmwq1qSCmllAsNBEopFeY0ECilVJjTQKCUUmFOA4FSSoU5DQRKKRXmNBAopVSY+/87pMH7Wke/6AAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x = np.linspace(-10,10,50)\n",
    "y1 = 1/(1+np.exp(0-x))-0.5\n",
    "y2 = x/(1+np.abs(x))\n",
    "y3 = (np.exp(x)-np.exp(0-x))/(np.exp(x)+np.exp(0-x))\n",
    "plt.title('activations')\n",
    "plt.plot(x,y1,marker='o')\n",
    "plt.plot(x,y2,marker='*')\n",
    "plt.plot(x,y3,marker='+')\n",
    "plt.legend(['sigmoid', 'softsign', 'tanh']) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 书面作业第2题\n",
    "用任何一种编程语言编程实现LMS学习算法，并将运行测试的结果抓图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "X = np.array([\n",
    "              [1,0,0],\n",
    "              [1,0,1],\n",
    "              [1,1,0],\n",
    "              [1,1,1],\n",
    "              [0,0,1],\n",
    "              [0,1,0],\n",
    "              [0,1,1],\n",
    "              [0,0,0]\n",
    "],dtype='float32')\n",
    "\n",
    "Y = np.array([\n",
    "              [-1],\n",
    "              [1],\n",
    "              [1],\n",
    "              [1],\n",
    "              [-1],\n",
    "              [-1],\n",
    "              [1],\n",
    "              [-1]\n",
    "],dtype='float32')\n",
    "\n",
    "\n",
    "class nueralnetwork:\n",
    "  def __init__(self, inputs=3, outputs=1, lr=0.1, epsilon=0.001):\n",
    "    self.weights = np.random.randn(inputs+1, outputs)\n",
    "    # self.bias = np.random.randn(outputs)\n",
    "    self.lr = lr\n",
    "    self.epsilon = epsilon\n",
    "\n",
    "  def hardlimit(self, v):\n",
    "    t = (v >= 0).astype('float32')\n",
    "    t = t*2-1\n",
    "    return t\n",
    "  \n",
    "  def forward(self, x, useact=True):\n",
    "    #增加一个新维度用来表示偏置值\n",
    "    n = np.ones((x.shape[0],1),dtype='float32')\n",
    "    x1 = np.concatenate((x,n),axis=1)\n",
    "    # print(x1)\n",
    "    if useact:\n",
    "        y = self.hardlimit(np.matmul(x1, self.weights))\n",
    "    else:\n",
    "        y = np.matmul(x1, self.weights)\n",
    "    return x1, y\n",
    "  \n",
    "  def train(self, x, y):\n",
    "    x1, out = self.forward(x,False)\n",
    "    loss = y - out\n",
    "    oldweights = self.weights\n",
    "    self.weights = self.weights + self.lr*np.matmul(np.transpose(x1), loss)\n",
    "    delta = np.linalg.norm(self.weights - oldweights)\n",
    "    return delta\n",
    "\n",
    "  def predict(self, x):\n",
    "    _, v = self.forward(x)\n",
    "    return v\n",
    "\n",
    "# 创新神经网络对象\n",
    "net = nueralnetwork()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0: delta: 1.507\n",
      "epoch 1: delta: 0.753\n",
      "epoch 2: delta: 0.386\n",
      "epoch 3: delta: 0.211\n",
      "epoch 4: delta: 0.131\n",
      "epoch 5: delta: 0.095\n",
      "epoch 6: delta: 0.077\n",
      "epoch 7: delta: 0.066\n",
      "epoch 8: delta: 0.058\n",
      "epoch 9: delta: 0.051\n",
      "[[-1.]\n",
      " [ 1.]\n",
      " [ 1.]\n",
      " [ 1.]\n",
      " [-1.]\n",
      " [-1.]\n",
      " [ 1.]\n",
      " [-1.]]\n"
     ]
    }
   ],
   "source": [
    "# 训练10个epoch\n",
    "for i in range(10):\n",
    "    print(f'epoch {i}: delta: {net.train(X,Y):.3f}')\n",
    "#测试一下输出\n",
    "print(net.predict(X))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 阅读作业\n",
    "\n",
    "我精读了一下本篇论文，对关键的公式做了推导。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 论文概述\n",
    "\n",
    "这篇论文从实验及理论推导两个方面讲了如何理解深度前馈神经网络为什么比较难训练，针对使用的不同激活函数（sigmoid, tanh, softsign）以及权值的初始化方法。  \n",
    "* 论文中发现Sigmoid/tanh激活函数不适合具有随机初始化的深度网络，它容易让后续的隐层进入饱和。  \n",
    "* softsign激活函数在对抗饱和问题上相比sigmoid要好，比较适合深度网络。  \n",
    "* 研究了训练时激活值梯度如何在不同层传递，当层间梯度雅可比矩阵的奇异值时远非1时，训练可能更困难\n",
    "* 提出了一种新的初始化方案（normalized initialization），它相比传统的初始化方法（standard initialization）可以更快地收敛。并通过实验验证了。\n",
    "\n",
    "### 关键解读\n",
    "\n",
    "对于几种激活函数的影响，主要是通过直觉观察+实验验证的方式，给出的结论就是sigmoid/tanh在避免后续层进入饱和上不理想。而softsign表现良好。\n",
    "\n",
    "重点讲一下理论推导权重参数的初始化方法。  \n",
    "在之前Bradley(2009)的研究中发现，从输出层向输入层后向传播的梯度在初始化之后会变小。在Bradley的研究中针对每层采用的是用线性激活函数，他发现后向传播梯度的方差在变小。我理解方差变小是网络饱和的表现。在这篇论文中也是用方差来表征的网络的饱和的。  \n",
    "* 我们希望在前向传递的过程中各层的输出值的方差保持不变，用数学形式表示如下($z^i$表示第$i$层的输出，$z^{i'}$表示第$i'$层的输出，$Var$表示方差)：\n",
    "\n",
    "$$\n",
    "\\forall(i,i'),\\; Var[z^i]=Var[z^{i'}] \\tag{8}\n",
    "$$\n",
    "\n",
    "上面这个式子，要这么理解，$z^i$是向量，$\\forall(i,j,i',j'),\\; Var[z_j^i]=Var[z_{j'}^{i'}]$\n",
    "\n",
    "* 我们还希望梯度在各层反向传递时方差保持不变，用数学形式表示如下($s^i=z^iW^i+b^i \\;and\\; z^{i+1}=f(s^i)$, $f$表示激活函数)：\n",
    "\n",
    "$$\n",
    "\\forall(i,i'),\\; Var[\\frac{\\partial Cost}{\\partial s^i}]=Var[\\frac{\\partial Cost}{\\partial s^{i'}}] \\tag{9}\n",
    "$$\n",
    "\n",
    "注意上面的公式中$\\frac{\\partial Cost}{\\partial s^i}$也是向量。  \n",
    "\n",
    "第(8)式很好理解，(9)式要解释一下。先看一下误差反向传播算法，每层的权值调整量如下：\n",
    "\n",
    "$$\n",
    "\\Delta W^i_{l,k}=-\\eta \\frac{\\partial Cost}{\\partial W^i_{l,k}}=-\\eta\\frac{\\partial Cost}{\\partial s_k^i}\\frac{\\partial s_k^i}{\\partial W^i_{l,k}}=-\\eta z_l^i\\frac{\\partial Cost}{\\partial s_k^i}\n",
    "$$\n",
    "\n",
    "从上面的式子可以看出，反向传播中，实际上传递的梯度是$\\frac{\\partial Cost}{\\partial s^i}$。\n",
    "\n",
    "在进一步推导之前，先说明一下：$Var[A]$表示什么意思(其中$A$是向量)？$Var[A]$是意思是$A$所有分量的方差相同，$Var[A]$表示它们共同的方差。\n",
    "\n",
    "那看看根据式(8)和(9)能得到啥呢？先看一下$Var[z^i]$:\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "Var[z^i] &=Var[f(z^{i-1}W^{i-1}+b^{i-1})] \\\\\n",
    "&=Var[f(z^{i-1}W^{i-1})]\\;\\;\\rightarrow论文中取b^i=0 \\\\\n",
    "&=Var[z^{i-1}W^{i-1}]  \\;\\; \\rightarrow论文中认为初始化阶段f'(s_k^i)\\approx 1, 即论文中式(4) \\\\\n",
    "&=Var[[\\sum_{j=1}^{n_{i-1}}z_j^{i-1}W_{j,0}^{i-1}, \\sum_{j=1}^{n_{i-1}}z_j^{i-1}W_{j,1}^{i-1},\\cdots,\\sum_{j=1}^{n_{i-1}}z_j^{i-1}W_{j,n_{i-1}}^{i-1}]] \\\\\n",
    "&=Var[\\sum_{j=1}^{n_{i-1}}z_j^{i-1}W_{j,0}^{i-1}] \\;\\;\\rightarrow向量中所有方差相同，我这里只取一个分量来表示一下 \\\\\n",
    "&=n_{i-1}Var[z^{i-1}W^{i-1}] \\;\\;\\rightarrow用向量整体来表征相同的个体 \\\\\n",
    "&=n_{i-1}Var[z^{i-1}]Var(W^{i-1}]+Var[z^{i-1}]E[W^{i-1}]^2+Var[W^{i-1}]E[z^{i-1}]^2\\\\\n",
    "&=n_{i-1}Var[z^{i-1}]Var[W^{i-1}] \\;\\;\\rightarrow取E(W^{i-1})=E(z^{i-1})=0，这里E表示均值，初始时选择满足均值为0 \\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "整理一下上面的式子，就得到了论文中的式(5)：\n",
    "\n",
    "$$\n",
    "Var[z^i] = Var[x]\\prod_{i'=0}^{i-1}n_{i'}Var[W^{i'}] \\tag{5}\n",
    "$$\n",
    "\n",
    "由式(5)和(8)可以得到：\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\forall i, Var[z^{i}]&=Var[z^{i-1}] \\\\\n",
    "\\forall i, n_{i}Var[W^i] &= 1 \\tag{10}\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "再来看一下$Var[\\frac{\\partial Cost}{\\partial s^i}]$，要注意$\\frac{\\partial Cost}{\\partial s^i}$是一个向量，这里我们考察一下它的一个分量$Var[\\frac{\\partial Cost}{\\partial s_k^i}]$:\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "Var[\\frac{\\partial Cost}{\\partial s_k^i}]&=Var[\\frac{\\partial Cost}{\\partial s^{i+1}}\\frac{\\partial s^{i+1}}{\\partial s_k^i}]\\\\\n",
    "&=Var[\\frac{\\partial Cost}{\\partial s^{i+1}}\\frac{\\partial (z^{i+1}W^{i+1}+b^{i+1})}{\\partial s_k^i}]\\\\\n",
    "&=Var[\\frac{\\partial Cost}{\\partial s^{i+1}}\\frac{\\partial (f(s^i)W^{i+1}+b^{i+1})}{\\partial s_k^i}]\\\\\n",
    "&=Var[\\frac{\\partial Cost}{\\partial s^{i+1}}\\frac{\\partial (f(s^i)W^{i+1})}{\\partial s_k^i}] \\;\\;\\rightarrow b^{i+1}是常数项，求导时为0\\\\\n",
    "&=Var[\\frac{\\partial Cost}{\\partial s^{i+1}}f'(s_k^i)W_{k,*}^{i+1}] \\;\\;\\rightarrow W_{k,*}^{i+1}表示W^{i+1}矩阵中第k行组成的行向量，对应论文中式(2) \\\\\n",
    "&=Var[\\frac{\\partial Cost}{\\partial s^{i+1}}W_{k,*}^{i+1}] \\;\\;\\rightarrow 论文中认为初始化阶段f'(s_k^i)\\approx 1, 即论文中式(4) \\\\\n",
    "&=Var[\\sum_{j=1}^{n_{i+2}}\\frac{\\partial Cost}{\\partial s_j^{i+1}}W_{k,j}^{i+1}] \\;\\;\\rightarrow W^{i+1}矩阵维度应该是n_{i+1} \\times n_{i+2}\\\\\n",
    "&=n_{i+2}Var[\\frac{\\partial Cost}{\\partial s_1^{i+1}}W_{k,1}^{i+1}] \\;\\;\\rightarrow 参考前面推导Var[z^i]相似过程\\\\\n",
    "&=n_{i+2}Var[\\frac{\\partial Cost}{\\partial s^{i+1}}]Var[W^{i+1}] \\;\\;\\rightarrow 参考前面推导Var[z^i]相似过程\\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "整理一下上面的式子，就得到了论文中的式(6)：\n",
    "\n",
    "$$\n",
    "Var[\\frac{\\partial Cost}{\\partial s^i}]=Var[\\frac{\\partial Cost}{\\partial s^d}]\\prod_{j=i+2}^d n_{j}Var[W^{j-1}] \\tag{6}\n",
    "$$\n",
    "\n",
    "这里的式(6)与论文中有稍许差别，感觉是论文中有错误。  \n",
    "由式(6)和(9)可以得到：\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\forall(i),\\; Var[\\frac{\\partial Cost}{\\partial s^i}]&=Var[\\frac{\\partial Cost}{\\partial s^{i+1}}] \\\\\n",
    "\\forall(i),\\; n_{i+1}Var[W^{i}]&=1 \\tag{11} \n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "由式(10)和(11)可以给出一个$Var[W^i]$的近似值：\n",
    "\n",
    "$$\n",
    "Var[W^i]=\\frac{2}{n_i+n_{i+1}}\\tag{12}\n",
    "$$\n",
    "\n",
    "式(12)的物理含义就是，如果每层的权值方差满足式(12)，那么前向传递的值以及后向传递的梯度的方差会保持稳定，不会饱和。我们可以反算一下什么样的均匀分布能够满足式(12)。假设对称的均匀分布：$U[-a,a]$，则$Var(U)=\\frac{(a-(-a))^2}{6}=\\frac{a^2}{3}$\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\frac{a^2}{3}&=\\frac{2}{n_i+n_{i+1}}\\\\\n",
    "a &= \\pm\\frac{\\sqrt{6}}{\\sqrt{n_i+n_{i+1}}}\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "因此，只要我们的权值满足如下均匀分布就可以上面式(12)的要求，这也是论文中提到的**normalized initialization**。\n",
    "\n",
    "$$\n",
    "W^i \\sim U\\Big[-\\frac{\\sqrt{6}}{\\sqrt{n_i+n_{i+1}}},\\frac{\\sqrt{6}}{\\sqrt{n_i+n_{i+1}}}\\Big] \\tag{16}\n",
    "$$\n",
    "\n",
    "再回过头来看看之前的文称之为**standard initialization**，也就是论文中的公式(1):\n",
    "\n",
    "$$\n",
    "W_{ij}\\sim U\\Big[-\\frac{1}{\\sqrt{n}},\\frac{1}{\\sqrt{n}} \\Big] \\tag{1}\n",
    "$$\n",
    "\n",
    "**standard initialization**下权值的方差，也很容易计算（假设所有层维度都是n）:\n",
    "$$\n",
    "Var[W]=\\frac{1}{3n}\\tag{15}\n",
    "$$\n",
    "\n",
    "根据式(15)，**standard initialization**会导致反向传递的梯度依赖于层的维度，同时可能会变小（饱和）。\n",
    "\n",
    "### 论文贡献\n",
    "\n",
    "1. 推导并给出了一种新的权值初始化方法，可以减少饱和，加快收敛。有理论，有实验。  \n",
    "2. 验证了三种激活函数（sigmoid,tanh, softsign）在深度神经网络中的效果，前两种不适合在深度网络中使用，softsign相对比较合适，这利益于softsign在靠近它的渐近线时比较平滑。  \n",
    "\n",
    "### 读后感悟\n",
    "\n",
    "1. 终于知道了所谓Xavier初始化方法的出处了。\n",
    "2. 论文中关于奇异值的论述还不是很明白。\n",
    "3. 论文中公式(12)只是一种选择，这里面感觉是可以有其他选择的，如果有时间是可以在这个方向上研究一下。\n",
    "\n",
    "### 扩展阅读\n",
    "\n",
    "1. Sigmoid激活函数非0均值会导致训练比较慢，原因与它会导致Hessian矩阵奇异值？这部分论文中提到了lecun的一篇1998年的论文，可以看一下。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "fb9f349487484364a76a86d689de2291b320d01c749a280950eb5136d479374c"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
